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Abstract 

This is an introductory survey, from a geometric perspective, on 
the Singular Value Decomposition (SVD) for real matrices, focusing 
on the role of the Terracini Lemma. We extend this point of view to 
tensors, we define the singular space of a tensor as the space spanned 
by singular vector tuples and we study some of its basic properties. 


1 Introduction 

The Singular Value Decomposition (SVD) is a basic tool frequently used in 
Numerical Linear Algebra and in many applications, which generalizes the 
Spectral Theorem from symmetric nx n matrices to general m x n matrices. 
We introduce the reader to some of its beautiful properties, mainly related to 
the Eckart-Young Theorem, which has a geometric nature. The implementa¬ 
tion of a SVD algorithm in the computer algebra software Macaulay2 allows 
a friendly use in many algebro-geometric computations. 

This is the content of the paper. In Section 2 we see how the best rank 
r approximation of a matrix can be described through its SVD; this is the 
celebrated Eckart-Young Theorem, that we revisit geometrically, thanks to 
the Terracini Lemma. In Section 3 we review the construction of the SVD 
of a matrix by means of the Spectral Theorem and we give a coordinate free 
version of SVD for linear maps between Euclidean vector spaces. In Section 
4 we dehne the singular vector tuples of a tensor and we show how they 
are related to tensor rank; in the symmetric case, we get the eigentensors. 
In Section 5 we dehne the singular space of a tensor, which is the space 
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containing its singular vector tuples and we conclude with a discussion of the 
Euclidean Distance (ED) degree, introduced hrst in j5]. We thank the referee 
for many useful remarks. 


2 SVD and the Eckart-Young theorem 


The vector space Ai = Airn,n oimxn matrices with real entries has a natural 
hltration with subvarieties Air = {'nr x n matrices of rank < r}. We have 

Ail d Ai2 d ... d AimirL{m,n} 

where the last subvariety Wlmin{m,n} coincides with the ambient space. 

Theorem 2.1 (Singular Value Decomposition) 

Any real m x n matrix A has the SVD 

A = DEV* 


where U, V are orthogonal (respectively of size m x m and n x n) and 
E = Diag(cTi, (72,...), with ai > a 2 > ... > 0. The m x n matrix E has zero 
values at entries (ij) with i ^ j and sometimes it is called pseudodiagonal 
(we use the term diagonal only for sguare matrices). 


The diagonal entries cij are called singular values of A and it is immediate 
to check that af are the eigenvalues of both the symmetric matrices AA* and 
A*A. We give a proof of Theorem 2.1 in ^ We recommend m for a nice 
historical survey about SVD. 

Decomposing E = Diag((Ji, 0, 0, ■ ■ ■ ) + Diag(0, < 72 , 0, •••) + ■■ ■ =: Si + S 2 + - ■ ■ 
we hnd 

A = UEiV* + UE 2 V* + ■■■ 

and the maximum i for which Uj 7 ^ 0 is equal to the rank of the matrix A. 


Denote by Uk, vi the columns, respectively, of U and V in the SVD above. 
From the equality A = DEV* we get AV = UE and considering the ith 
columns we get 

Avi = (AV)i = {UE)i = {{ui, ■ ■ • , Um)V)iag{ai,a2, •■■))* = 
while, from the transposed equality A* = VE*U*, we get A*Ui = aiVi. 

So if 1 < i < min{m,n{, the columns Ui and Vi satisfy the conditions 

Avi = (7jMj and A*Ui = OiVi. (1) 
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Definition 2.2 The pairs {ui,Vi) in 0 are called singular vector pairs. 

More precisely, if 1 < i < min{m,n}, the vectors Ui and Vi are called, 
respectively, left-singular and right-singular vectors for the singular value ai. 

If the value cxj appears only once in S, then the corresponding pair (wj, Vi) 
is unigue up to sign multiplication. 


Remark 2.3 The right-singular vectors corresponding to zero singular val¬ 
ues of A span the kernel of A; they are the last n — rk{A) columns ofV. 

The left-singular vectors corresponding to non-zero singular values of A 
span the image of A; they are the first rk{A) columns ofU. 


Remark 2.4 The unigueness property mentioned in Definition fS.Sl shows 
that SVD of a general matrix is unigue up to simultaneous sign change in 
each pair of singular vectors Ui and Vi. With an abuse of notation, it is 
customary to think projectively and to refer to “the” SVD of A, forgetting 
the sign change. See Theorem 3^ for more about unigueness. 

For later use, we observe that = aiUi ■ v\. 


Let II — II denote the usual V norm (called also Frobenius or Hilbert- 
Schmidt norm) on JVl, that is VA G M. 

\\A\\ := ^Jtr{AA^) = ^ij- Note that ii A = UTV^, then \\A\\ = 

ITWl 


The Eckart-Young Theorem uses SVD of the matrix A to find the matrices 
in which minimize the distance from A. 


Theorem 2.5 (Eckart-Young, 1936) 

Let A = UTV^ be the SVD of a matrix A. Then 

• UTiV* is the best rank 1 approximation of A, that is 
11^4 — t/SiV*| I < I |A — X| I for every matrix X of rank 1. 

• For any 1 < r < rank{A), f/SiV* + ... + UTAV^ is the best rank r 
approximation of A, that is \ \A — UTiV* — ... — UTjAV^W < HA — X|| 
for every matrix X of rank < r. 


Among the infinitely many rank one decompositions available for matri¬ 
ces, the Eckart-Young Theorem detects the one which is particularly nice 
in optimization problems. We will prove Theorem 2.5 in the more general 
formulation of Theorem 12.91 
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2.1 Secant varieties and the Terracini Lemma 

Secant varieties give basic geometric interpretation of rank of matrices and 
also of rank of tensors, as we will see in section 

Let X C Pld be an irredncible variety. The fc-secant variety of X is 
defined by 

(Tk{X) := IJ FSpan{pi,... ,pk} (2) 

pi,...,Pk&X 

where FSpan {pi,... ,pk\ is the smallest projective linear space containing 
Pi,... ,Pk and the overbar means Zariski closnre (which is equivalent to Eu¬ 
clidean closure in all cases considered in this paper). 

There is a filtration X = ai{X) G a 2 {X) C ... 

This ascending chain stabilizes when it fills the ambient space. 

Example 2.6 (Examples of secant varieties in matrix spaces.) We may 

identify the space M of m x n matrices with the tensor product MW 0 M”. 
Hence we have natural inclusions Air C ® M”. Since Air o,re cones, 
with an abuse of notation we may call with the same name the associated 
projective variety Air C P(M'" 0 M"^). The basic equality we need is 

ar{Aii) = Air 

which corresponds to the fact that any rank r matrix can be written as the 
sum of r rank one matrices. 

In this case the Zariski closure in is not necessary, since the union is 
already closed. 

The Terracini Lemma (see [9] for a proof) describes the tangent space T 
of a fc-secant variety at a general point. 

Lemma 2.7 (Terracini Lemma) 

Let z G MSpan {pi,... ,pk} be general. Then 

^z<7k{A) = FSpan{Tp^X ,... ,Tp^T} . 

Example 2.8 (Tangent spaces to Air) The tangent space to Aii at a 

point u®v is 0 n -f -u 0 M” .■ 

any curve 7 (f) = u{t) 0 v{t) in Aii with 7 ( 0 ) = u 0 n has derivative for 
f = 0 given by ■u'(O) 0 n-l--u 0 n'(O) and since M'(0),n'(0) are arbitrary vectors 
in respectively, we get the thesis. 
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As we have seen in Example \2.6 , the variety A4r can be identified with 
the r-secant variety of Aii, so the tangent space to Air at a point 
U(J2i + ■ ■ ■ + 'Er)V^ can he described, by the Terracini Lemma, as 

^UT,iV^Aii + ■ ■ ■ + TT^/s^ytA^i = “1“ ■ ■ ■ arUr(Slvt-^^l 


^v\ +Ui 


+ ■■■ + 


^v; + Ur 


2.2 A geometric perspective on the Eckart-Young The¬ 
orem 


Consider the variety Air C ® ME of matrices of rank < r and for any 
matrix A G M'" ® M" let dA{—) = d{A,—): Air —)■ M be the (Euclidean) 
distance function from A. If rkA > r then the minimum on Air of dA is 
achieved on some matrices of rank r. This can be proved by applying the 
following Theorem 2^ to Air' for any r' < r. Since the variety Air is singular 
exactly on Air-i, the minimum of dA can be found among the critical points 
of dA on the smooth part Air \ Air-i- 


Theorem 2.9 (Eckart-Young revisited) Example 2.3] 

Let A = UEV^ be the SVD of a matrix A and letl <r < rk(A). All the crit¬ 
ical points of the distance function from A to the (smooth) variety Air\Air-i 
are given by +... + Sj^) Y, where Sj = Diag(0,..., 0, Uj, 0,..., 0), with 
1 <i < rk{A). If the nonzero singular values of A are distinct then the num¬ 
ber of critical points is . 


Note that LfHiV*' are all the critical points of the distance function from 
A to the variety Ali of rank one matrices. So we have the important fact 
that all the critical points of the distance function from A to Ali allow to 
recover the SVD of A. 

For the proof of Theorem 2.9 we need 


Lemma 2.10 If Ai = ui (^vi, A 2 = U 2 ®V 2 are two rank one matrices, then 
< Ai,A 2 > = < Ui,U2 ><Vi,V2 >■ 


Proof. < ^1,^42 >= tr{AiAl) = 



( «„ ^ 

\ 

/ 

'^M21 ^ 

\ 

tr[ 


■ (Mii, • • • ,Min) 



■ (m21, • • • , M2m) 


y y 'a\m j 

) 

\ 

\v2n) 

/ 


(Sfc 'aikV2k) U2i — 'aiiU2i Yhk —< Ml, ^2 >< Ml, M 2 >. 
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Lemma 2.11 Let B E Ai. If < B, MB' ® v >= 0, then < Row{B),v >= 0. 
If < B,u ^M^ >= 0, then < Co\{B),u >= 0. 

Proof. Let {ei, ■ • • , Cm} be the canonical basis of M™; then, by hypothesis, 

< B,ek<^v >= 0 \/k = 1, ■ ■ ■ , m. We have 0 = tr [B{v^ ® e|)] = 

tr [-8(0, • • • , 0, n, 0, • • • ,0)] =< B’^, v >, where B^ denotes the fcth row of B, 

so that the space Row(-B) is orthogonal to the vector v. 

In a similar way, we get < Co1(-B),m >= 0. 


By nsing Terracini Lemma 2.7 we can prove Theorem 2.9 


Proof of Theorem 12.91 

The matrix U + ■ ■ ■ + is a critical point of the distance fnnction 

from A to the variety Air if and only if the vector A — + ■ ■ ■ + 

is orthogonal to the tangent space [see 

) H-h (M”" (g) vj + Ui^ (g M*"). 

+ --- + S,JW)=V(S,, + ■■■ + 


T 


tAir — 


{/(Si,+...+Sigy‘VW^ = g vf + Ui, 

From the SVD of A we have A — {U{Ri 


P‘ji)V^ = CTjj^Uj.^ + ■ ■ ■ +ajjUjj gn*^ where {ji, • • ■ , jf} is the set of indices 
given by the difference {1, ■ ■ ■ , r/c(y4)}\{zi, ■ ■ ■ , v}- 
Let {ei, • • ■ , Cm} be the canonical basis of M 
< ® e; g vl >= aj^ < Uj^, ei XI 

distinct colnmns of the orthogonal matrix V. So the matrices UTij^V^ are 


By Lemma 2.10 we get: 


>= 0 since are 


orthogonal to the spaces g v. 




In a similar way, since U is an orthogonal matrix, the matrices UY^j^V^ 
are orthogonal to the spaces g M”. So A — ( 8 (Sj^ + ■ ■ ■ + is 

orthogonal to the tangent space and + ■ ■ ■ + is a critical point. 

Let now B G Air be a critical point of the distance fnnction from A to 
Air- Then A — B is orthogonal to the tangent space T^Al^- 

Let B = U'{E[ + ■■■ A-B = U"{E'I + ■■■ S")W'* be SVD of B 

and A — B respectively, with ^ 0 and S'/ ^ 0. 

Since A — B is orthogonal to T^Alr = (R"* ® v'l + u[ 

R""), by Lemma 2.11 we get < Col{A - 


^ y/t 


g M") + • • • + 

_ B)A'k >= 0 

and < Row{A — -B),n/ >= 0 A: = 1, • • ■ , r. In particnlar, Col{A — B) is a. 
vector snbspace of Span{u[, • • • , u'^}^ and has dimension at most m — r while 
Row{A — B) is a vector snbspace of Span{v[, ■ ■ ■ and has dimension 

at most n — r, so that I < min{m, n} — r. 

From the eqnality A — B = («'/,..., m/, 0 ..., 0) (S'/ + ■ ■ ■ S/)V"* we get 
Col{A — B) <Z Span{u'{, ..., u”} and eqnality holds by dimensional reasons. 
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In a similar way, Row{A — B) = Span{v'l, ■ ■ ■ ,n"}. This implies that 
the orthonormal columns , m", u[, ■ ■ ■ , can be completed with or¬ 

thonormal m — l — r columns of M"* to obtain an orthogonal mxm matrix U, 
while the ortho normal columns Vi, - ■ ■ , v'j. can be completed with 

orthonormal n — I — r columns of M” to obtain an orthogonal n x n matrix 
1 /. 

/S"00\ /000\ 

We get A — B = U i OOoj W, B = U i 0 S' 0 1 V^, where 
\000/ \0 00/ 

S" = Diag((T'/,..., a'/) and S' = Diag((T^,..., cr^). 

/S" 0 0\ 

So A = {A — B) + B = U { 0 S' 0 1 1/^ can easily be transformed to 

\ 0 0 0/ 

a SVD of A by just reordering the diagonal elements crfs and ct"’s and the 
critical point B is of the desired type. □ 


The following result has the same flavour of Eckart-Young Theorem 2.9 


Theorem 2.12 (Baaijens, Draisma) /H Theorem 3.2] 

Let A = UYA/^ he the SVD of a n x n matrix A. All the eritical points of 
the distance function from A to the variety 0{n) of orthogonal matrices are 
given by the orthogonal matrices 17Diag(±l,..., ±1)W and their number is 
2 ”. 


Actually, in [T], the result is stated in a slightly different form, which is 
equivalent to this one, that we have chosen to make more transparent the 
link with SVD. It is easy to check that, among the critical points computed in 
Theorem 2.12 the one with all plus signs, corresponding to the orthogonal 
matrix W', gives the orthogonal matrix closest to A. This is called the 
Lowdin orthogonalization (or symmetric orthogonalization) of A. 


3 SVD via the Spectral Theorem 

In this section we prove Theorem |2.1| as a consequence of the Spectral The¬ 
orem. We recall 

Theorem 3.1 (Spectral Theorem) 

For any symmetric real matrix B, there exists an orthogonal matrix V such 
that V~^BV = V^BV is a diagonal matrix. 
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Remark 3.2 Since the Euclidean inner product is positive definite, it is el¬ 
ementary to show that for any real m x n matrix A we have Ker{A^A) = 
Ker{A) and Ker{AA^) = Ker{A*). 


Proof of Theorem 12.11 

Let A be an m X n matrix with real entries. The matrix A'^A is a symmetric 
matrix of order n and it’s positive semidehnite. By the Spectral Theorem, 
there exists an orthogonal matrix V (of order n) snch that 


V-\A^A)V = V\A^A)V = 


D 0 
0 0 


where D is diagonal of order r = ik{A^A) = ik{A) (see Remark 3.2) and is 
positive definite: D = Diag((ii, • • ■ , dr) with di > d 2 > ■ ■ ■ dr > 0. 

Let fi, • • • , be the colnmns of V; then 


{A^A){vi,--- ,Vn) = (ni, 


D 0 


, 1 0 0 ) ^ (c^Rl, • • ■ , drVr, 0, • • ■ ,0) 


G Ker{A^A) = Ker{A) (see Remark 3.2). 


and Vr+i, ■ ■ ■ ■ 

Let CTj = y/di, i = 1, ■ ■ ■ ,r and let Ui = {l/afiAvi G M™'. These vectors 
are orthonormal since < Ui,Uj >= < Avi,Avj >= —< Vi,A^Avn >= 

< Vi,djVj >= ^ < Vi,Vj >= ffidij. Thns it’s possible to find m — r or¬ 
thonormal vectors in snch that the matrix U := (mi, • • • , Ur, Mr+i, • • • , Mm) 

f 0\ 

is an m X m orthogonal matrix. Define S:=( ^ Qjtobeanmxn matrix 

with m — r zero rows, = Diag((Ji, ■ ■ ■ , cxr). Then 


f/SR* = ( - Avi, ■ • ■ , - AVr, Mr+l, ' ' ' ,Ur. 

\ai ar 






(JrVl 

0 

= A{vi,-- 

, Vr) 


V 0 ) 



U/ 


Since V is orthogonal we have 

/ v\\ 


4 = (mi • • • Vn) 


= {Vi ■■■ Vr) 


\</ 


( 


VrJ 


(n^+i • •• Vn) 


y+i\ 


< / 
















Hence we get 




A 

f 

■ Vn) 


\ 


V 


\< J 

/ 


A 


since Vr+i, ■ ■ ■ ,Vn G Ker{A). 


□ 


Lemma 3.3 

• (1) Let af > ■ ■ ■ > al > 0 be the distinct non zero eigenvalues of A^A 
and Vi = Ker{A^A — afln) he the corresponding eigenspaces, 

Ho = Ker{A^A) = Ker{A). Then 

= (©iiH) © Ho 

is an orthogonal decomposition of ML. 

• (2) Let Ui = Ker{AA^ - afim) , Uq = Ker{AA^) = Ker{A^). Then 

AVi = Ui and A^Ui = H */* = !,•■■ A- 

• (3) = (©tif^*) © Uo 

is an orthogonal decomposition of and af > ■ ■ ■ > al > 0 are the 
distinct non zero eigenvalues of AA^. 

• (4) The isomorphism —A|vi : H —t Ui is an isometry with inverse 

}-A%, : U, V,. 

Proof. (1) is the Spectral Theorem. In order to prove (2), AVi © Ui since 
Vtc G H one has {AA^){Aw) = A{A^A)w = afAw. In a similar way, A^Ui C 
H. On the other hand, Vz G Ui one has 2; = \{AA^)z = A{\A^z) G AVi so 

^ i 

that AT/i = Ui- In a similar way, A^Ui = H- (3) and (4) are immediate from 

( 2 ). 


Lemma 3.3 may be interpretated as the following coordinate free version 
of SVD, that shows precisely in which sense SVD is unique. 


9 







Theorem 3.4 (Coordinate free version of SVD) 

Let V, U he real vector spaces of finite dimension endowed with inner 
products <, >v and <, >u and let F: V ^ U be a linear map with adjoint 
: U —)■ V, defined by the property < Fv,u >u=< v,F^u >v Vu G V, Vm G 
U. Then there is a unique decomposition (SVD) 


k 

F = ^ ^iFi 

i=l 

with (Ji > ... > (jfc > 0, Fj: V —> W linear maps such that 

• FiFj and F(Fj are both zero for any i j, 

• Fi\im{F*)'- Irn{F() — Im{Fi) is an isometry with inverse F(. 

Both the singular values dj and the linear maps Fi are uniquely determined 
from F. 


By taking the adjoint in Theorem 3.4, F* = is the SVD of F' 


The first interesting conseqnence is that 


FF‘ = J2<’IrF‘ and F‘F = J2<^1F‘F 


2=1 


2 = 1 


are both spectral decomposition (and SVD) of the self-adjoint operators 
FF* and F*F. This shows the nniqneness in Theorem 3.4 Note that 


V = (^(Bi=iIm{F()) 0FerF and U = (©f^i/m(Fj)) 0FerF* are both or¬ 
thogonal decompositions and that rkF = XliLi rkFj. 

Moreover, F"*" = Yli=i is the Moore-Penrose inverse of F, express¬ 

ing also the SVD of F+. 

Theorem 3.4 extends in a straightforward way to hnite dimensional com¬ 
plex vector spaces V and U endowed with Hermitian inner products. 


4 Basics on tensors and tensor rank 

We consider tensors A G © ... ® K^d+i where F = M or C. It is 

convenient to consider complex tensors even if one is interested only in the 
real case. 
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Figure 1: The visualization of a tensor in 0 0 K^. 


Entries of A are labelled by d indices as 

For example, the expression in coordinates of a 3 x 2 x 2 tensor A as in 
Figure [T] is, with obvious notations, 

^ = (^oaoXoVo^Q + o,ooiXoyoZi + cloioXqI/iZq + aQiiXoUiZiA 
O'IooXiUqZq + aiQiXiHQZi + anoXiyiZo + aiiiXiyiZi+ 

02002^22/02^0 + 02012^22/o 2^1 + 02102^22/12^0 + 02lia;22/l2;i. 


Definition 4.1 A tensor A is decomposable if there exist x^ G fori = 

1 ,..., d, such that = xl^xf^ ■ ■ - ^fd- equivalent way, A = x^®.. .®x'^. 

Dehne the rank of a tensor A as the minimal number of decomposable 
summands expressing A, that is 


rk(y4) := min{r|y4 = '‘^^Ai^Ai are decomposable} 

i=l 

For matrices, this coincides with usual rank. For a (nonzero) tensor, 
decomposable rank one. 

Any expression A = X]i=i with Ai decomposable is called a tensor 
decomposition. 

As for matrices, the space of tensors of format (rii + 1 ) x ... x {nd + 1 ) 
has a natural hltration with subvarieties 

Tr = {A E ® ® rank (A) < r}. We have 

Ti C r2 C ... 

Corrado Segre in XIX century understood this hltration in terms of pro¬ 
jective geometry, since 7} are cones. 

The decomposable (or rank one) tensors give the “Segre variety” 
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7 i ~ X ... X P"''* c ® ® 

The variety Tk is again the fc-secant variety of 7i, like in the case of 
matrices. 


For K = M., the Euclidean inner product on each space induces the 

inner product on the tensor product (compare with Lemma 

• ■<S)Xd\\'^ = 


2 . 10 ) 


With respect to this product we have the equality ||a;i 


A best rank r approximation of a real tensor A is a tensor in 7^ 
which minimizes the /^-distance function from A. We will discuss mainly the 
best rank one approximations of A, considering the critical points T G 7i for 
the /^-distance function from A to the variety 71 of rank 1 tensors, trying to 
extend what we did in ^ The condition that T is a critical point is again 
that the tangent space at T is orthogonal to the tensor A — T. 


Theorem 4.2 (Lim, variational principle) fl^ 

The critical points xi ® ® G 71 of the distance function from A G 

]^ni+i 0 _ _ _ 0 to the variety 7 i of rank 1 tensors are given by d-tuples 

(xi,..., Xd) G X ... X such that 

A ■ {xi ® ... Xi... ® Xd) = Axj Vi = 1,..., 7 (3) 

where A G M, the dot means contraction and the notation Xi means that Xt 
has been removed. 

Note that the left-hand side of (|^ is an element in the dual space of 
l^rii+i^ so in order for ([^ to be meaningful it is necessary to have a metric 
identifying with its dual. We may normalize the factors Xi of the tensor 
product a;i ® ® in such a way that ||a:j|p does not depend on i. Note 

that from ([^ we get A • (a:i ® ® Xd) = A| |a:j| p. Here, (a;i,..., Xd) is called 

a singular vector d-tuple (dehned independently by Qi in [ 12 ]) and A is called 
a singular value. Allowing complex solutions to (|^ , A may be complex. 


Example 4.3 We may compute all singular vector triples for the following 
tensor in 0 0 


/ =6x01/020 

- 2014x01/120 
+ 48x01/2 20 

- 31x01/021 
+ 63x01/121 

- 3X01/221 


+ 2X11/020 + 6X21/020 
+ 121X11/120 - 11X21/120 
-13x11/220 - 40x21/220 
+93x11/021 + 97x21/021 
+41x1^12:1 — 9Ax2yizi 
-\-47xiy2Zi -\-4:X2y2Zi 
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We find 15 singular vector triples, 9 of them are real, 6 of them make 3 
conjugate pairs. 

The minimum distance is 184.038 and the best rank one approximation is 
given by the singular vector triple 

{xo - .0595538X1 + .00358519x2)(j/o - 289.637yi + 6.98717y2)(6.95378zo - .20796872i). TcnSOr 
decomposition of f can be computed from the Kronecker normal form and 
gives f as sum of three decomposable summands, that is 

f = (.450492a!o - 1.43768a:i - 1.409252!2)(-.923877j/o - .986098i/i - .6465S4y2)(.S097772o + 68.2814zi) + 
(-.582772a!o + .548689i:i + 1.93447x2)(.14885l!/o - 3.43755!;i - 1.07165 h2)(18.6866zo + 28.1003zi) + 

(1.06175a!o - .0S02873a:i - .0580488a:2)(-.0125305!/o + 3.22958j/i - .0575754j/2)(-598.154zo + 10.8017zi) 

Note that the best rank one approximation is unrelated to the three sum¬ 
mands of minimal tensor decomposition, in contrast with the Eckart- Young 
Theorem for matrices. 

Theorem 4.4 (Lim, variational principle in symmetric case) fWf The 

critical points of the distance function from A G Sym'^M”+^ to the variety Ti 
of rank 1 tensors are given by d-tuples x'^ G such that 

A ■ = Ax. (4) 

The tensor x in Q is called a eigenvector, the corresponding power x'^ is 
a eigentensor , X is called a eigenvalue. 

4.1 Dual varieties and hyperdeterminant 

If T C PI/ then 

X* := {H G Pl/*|3 smooth point p E X s.t. TpX C Hj 

is called the dual variety of X (see [3 Chapter 1]). So X* consists of hyper¬ 
planes tangent at some smooth point of X. 

In Enclidean setting, dnality may be nnderstood in terms of orthogonality. 
Considering the affine cone of a projective variety X, the dnal variety consists 
of the cone of all vectors which are orthogonal to some tangent space to X. 

Let m < n. The dnal variety of m x n matrices of rank < r is given by 
m X n matrices of rank <m — r ([3 Prop. 4.11]). In particnlar, the dnal of 
the Segre variety of matrices of rank 1 is the determinant hypersnrface. 
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Let Hi < ... < rid. The dual variety of tensors of format (ni + 1) x 
... X {jid + 1) is by definition the hyperdeterminant hypersurface, whenever 
Ud < Eti rii. Its equation is called the hyperdeterminant. Actually, this 
dehnes the hyperdeterminant up to scalar multiple, but it can be normalized 
asking that the coefficient of its leading monomial is 1 . 


5 Basic properties of singular vector tuples 
and of eigentensors. The singular space of 
a tensor. 


5.1 Counting the singular tuples 


In this subsection we expose the results of [B] about the number of singular 
tuples (see Theorem 4.2 


of a general tensor. 


Theorem 5.1 The number of (complex) singular d-tuples of a general 
tensor t G ® ® is egual to the coefficient of n^=i 

polynomial 


d -ni+l 

nV 

i=i 


t 


rii+l 


where U = tj ■ 


Amazingly, for d = 2 this formula gives the expected value min(?7,i + l, n 2 + l). 

For the proof, in [6] the d-tuples of singular vectors were expressed as 
zero loci of sections of a suitable vector bundle on the Segre variety 7i. 

Precisely, let 7i = x ... x P(C”''^'''^) and let Tr*: 7i —)■ P(C”‘’''^) 

be the projection on the i-th factor. Let 0(1,..., 1) be the very ample line 

d 

bundle which gives the Segre embedding and let Q be the quotient bundle. 
Then the bundle is i'^iQ) © 0( 1 , ... , 1 , 0 ,1,..., 1). 

t 

i 


The top Chern class of this bundle gives the formula in Theorem |5.1 
In the format (2,..., 2) the number of singular d-tuples is d!. 
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The following table lists the number of singular triples in the format 
(hi, ^2, ha) 


hi, h2, hs 

c(hi,h2,h3) 


2,2,2 

6 


2,2,n 

8 

n>3 

2,3,3 

15 


2,3, n 

18 

n > 4 

2, n, n 

n{2n — 1) 


3,3,3 

37 


3,3,4 

55 


3,3, n 

61 

n > 5 

3,4,4 

104 


3,4,5 

138 


3,4, n 

148 

n>6 


The number of singular h-tuples of a general tensor A G 0 ... 0 

when ni,...,nd-i are hxed and ri d in creases, stabilizes for rid > 


Yli=i it can be shown from Theorem 


5.1 


For example, for a tensor of size 2 x 2 x n, there are 6 singular vector 


triples for n = 2 and 8 singular vector triples for n > 3. 


The format with rid = is the boundary format, well known in 

hyperdeterminant theory [7]. It generalizes the square case for matrices. 

The symmetric counterpart of Theorem 5.1 is the following 


Theorem 5.2 (Cartwright-Sturmfels) The number of (complex) eigen- 
tensors of a general tensor t G P(Sym'^M’^+^) is egual to 

(h- 1)’"+^ - 1 
h- 2 ■ 


5.2 The singular space of a tensor 


We start informally to study the singular triples of a 3-mode tensor A, later 
we will generalize to any tensor. The singular triples x 0 y 0 z oi A satisfy 
(see Theorem 4.2 ) the equations 


ioiU 
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hence, by eliminating A, the equations (for every k < s) 






*0,*i 


which are linear equations in the Segre embedding space. These equations 
can be permuted on x, y, z and give 


{ Sio,ii i^iohkXioVhZs - Ai^i^sXioVij^Zk) = 0 for 0 < /c < s < ns 

Eio,i 2 i^ioki 2 ^ioysZi 2 - Ai^shXioVkZi^) = 0 foT 0 < k < s < n 2 (5) 

Eii,i 2 {Aki.i^XsVi^Zi^ - Asi^i^XkVnZi^) = 0 for 0 < /c < s < ni 

These equations dehne the singular space of A, which is the linear span 

of all the singular vector triples of A. 

The tensor A belongs to the singular space of A, as it is trivially shown 
by the following identity (and its permutations) 


E 




^0^1S 


AiQij^sA{g{-^k') 0 . 


In the symmetric case, the eigentensors x'^ of a symmetric tensor 
A G Sym'^C"’’*'^ are dehned by the linear dependency of the two rows of 
the 2 X (n + 1) matrix 


Taking the 2x2 minors we get the following 


Definition 5.3 If A G Sym'^C"'"''^ is a symmetric tensor, then the singular 
space is given by the following ("'^^) linear eguations in the unknowns x'^ 

dA{x^~^) dA{x'^~^) 

X Xi Xj 0 

OXj OXi 

It follows from the dehnition that the singular space of A is spanned by 
all the eigentensors x*^ of A. 


Proposition 5.4 The symmetric tensor A G Sym'^C"’*'^ belongs to the sin¬ 
gular space of A. The dimension of the singular space is — (” 2 ^) ■ The 
eigentensors are independent for a general A (and then make a basis of the 
singular space) just in the cases Sym'^C^, Sym^C"'’''^ Sym^C^. 
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Proof. To check that A belongs to the singular space, consider dual variables 
Vi = Then we have (^yi - ■ Aix) = ^ ■ jA - jA ■ which 

dxj \ayj^ ayi^-'J ' ■' oyj dxi dyi dxj^ 

vanishes by symmetry. To compute the dimension of the singular space, 
hrst recall that symmetric tensors in Sym'^C”’*'^ correspond to homogeneous 
polynomials of degree d in n + 1 variables. We have to show that for a general 
polynomial A, the ("' 2 ^) polynomials §A^i~ ^ < J independent. 

This is easily checked for the Fermat polynomial A = YA=o A d > 3 and 
for the polynomial A = J2p<q^p^q for d = 2. The case listed are the ones 
where the inequality 


(d- 1)^+^ - 1 
d- 2 


> 


n + d 
d 


n + 1 
2 


is an equality (for d = 2 the left-hand side reads as YA=oA — 1)* = '^ + !)• 
Denote by ej the canonical basis in any vector space C"". 

Proposition 5.5 Let rii < ... < Ud ■ If A E ® ... <8) is a 

tensor, then the singular space of A is given by the following YA=i 
linear eguations. The AA^) egwafions of the i-th group (i = l,...,dj for 
x‘^ are 

A ( /Y»l p, ^d\ ( A ( /y»2 p, rpd\ f — O 

,0/ , . . . , Cp , . . . , a. ) fq jn.yjj , u./ , . . . , , . . . , O/ y u 

t t 

i i 


for 0 < p < q < Ui. The tensor A belongs to this linear space, which we call 
again the singular space of A. 


Proof. Let A = ® ... ® xA Then we have for the hrst 

group of equations 

^ ^ Ak,i2,...,id^s,i2,...,id ~ I^s,i2,...,id^k,i2,...,id) ~ ^ 0 < /c < S < Ui 

*2,■■■,*<; 


and the same argument works for the other groups of equations. □ 

We state, with a sketch of the proof, the following generalization of the 


dimensional part of Prop. 5.4 
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Proposition 5.6 Let ni < ... < and N = ni=i + !)■ 

If A E 0 ... ® is a tensor, the dimension of the singular space 

of A is 

j nti(". + 1) - Et. ("•?') forn,+ l<N 

1 rE) - Et; (T) forn, + l>N. 

The singular d-tuples are independent (and then make a basis of this space) 
just in cases d = 2, ® ® C” for n > 4. 

Proof. Note that if + 1 > iV, for any tensor A G ® ® 

there is a subspace L C of dimension N such that A G ® 

... ® c«d-i+i (g) hence all singular d-tuples of A lie in A G ® 

... ® £na-i+i 0 Note that for + 1 = Ilti + !)> ^^en the singu¬ 
lar space has dimension N = nf=i(^* + 1) ~ Sf=i be shown 

that posing k{ii,... ,id-i) = Yl'jZl (ns<j-i(^^ + 1 )) p > ^ben the tensor 

ni »^d-l 

A = k{H,id-i)el ... eYYk{h,...M.^) is general in the sense 

P =0 id-l =0 

that the ("^* 2 ^^) corresponding equations are independent (note that 

k{ii,, id-i) covers all integers between 0 and iV — 1). For -|- 1 > iV the 
dimension stabilizes to iV^ - ~ {^ 2 ^) = {^ 2 ^) ~ Ylt=i ^" 2 ^) ■ 

Remark 5.7 For a general A G ® ® the linear 

eguations of the singular space of A are independent if Ud + I < N + 1. 

Remark 5.8 In the case of symmetric matrices, the singular space of A con¬ 
sists of all matrices commuting with A. If A is regular, this space is spanned 
by the powers of A. If A is any matrix (not necessarily symmetric), the sin¬ 
gular space of A consists of all matrices with the same singular vector pairs 
as A. These properties seem not to generalize to arbitrary tensors. Indeed 
the tensors in the singular space of a tensor A may have singular vectors 
different from those of A, even in the symmetric case. This is apparent for 
binary forms. The polynomials g having the same eigentensors as f, satisfy 
the eguation pxy — gyX = \{fxy — fyx) for some \, which in degree d even 
has (in general) the solutions g = p.if -E p. 2 {x'^ + with pi,/i 2 G C, while 

for degree d odd has (in general) the solutions g = pif. In both cases, these 
solutions are strictly contained in the singular space of f. 
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In any case, a positive result which follows from Prop. 5.4, 5.5, 5.6 is the 
following 


Corollary 5.9 

• (i) Let ni < ... < Ud, N = Ilti + 1), M = min(iV,+ 1). A 

general tensor A G ® ® has a tensor decomposition 

given by NM — ~ {^ 2 ^) singular vector d-tuples. 

• (a) A general symmetric tensor A G Sym'^C”^^ has a symmetric tensor 

decomposition given by eigentensors. 

The decomposition in (i) is not minimal unless d = 2, when it is given by 
the SVD. 

The decomposition in (ii) is not minimal unless d = 2, when it is the 
spectral decomposition, as sum of {n + 1) (sguared) eigenvectors. 


5.3 The Euclidean Distance Degree and its duality prop¬ 
erty 

The construction of critical points of the distance from a point p, can be 
generalized to any affine (real) algebraic variety At. 

Following |5], we call Euclidean Distance Degree (shortly ED degree) the 
number of critical points oi dp = d{p,—): At —)■ M, allowing complex solu¬ 
tions. As before, the number of critical points does not depend on p, provided 
p is generic. For a elementary introduction, see the nice survey [T3] . 

Theorem 2.9| says that the E D degree of the variety AAr dehned in ^ is 


mm{m,n}^^ while Theorem 2.12|says that the ED degree of the variety 0{n) 
is 2'^. The values computed in Theorem |5.1| give the ED degree of the Segre 


variety x ... x while the Cartwright-Sturmfels formula in Theorem 


5.2 gives the ED degree of the Veronese variety ^^(P"'). 


Theorem 5.10 |3 Theorem 5.2, Corollary 8.3] Let p be a tensor. There is 
a canonical bijection between 

• critical points of the distance from p to rank < 1 tensors 


• critical points of the distance from p to hyperdeterminant hypersurface. 


Correspondence is x p — x 
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In particular, from the 15 critical points for the distance from the 3x3x2 


tensor / dehned in Example |4.3| to the variety of rank one matrices, we may 
recover the 15 critical points for the distance from / to the hyperdeterminant 
hypersurface. It follows that Det(/ — Pi) = 0 for the 15 critical points p*. 


The following result generalizes Theorem 5.10 to any projective variety X 


Theorem 5.11 |3 Theorem 5.2] Let X C 
There is a canonical bijection between 


be a projective variety, p G P" 


• critical points of the distance from p to X 

• critical points of the distance from p to the dual variety X*. 
Correspondence is x p — x. In particular EDdegree(T) = EDdegree(T*) 



Figure 2: The bijection between critical points on X and critical points on 


5.4 Higher order SVD 

In [3], L. De Lathauwer, B. De Moor, and J. Vandewalle proposed a higher 
order generalization of SVD. This paper has been quite influential and we 
sketch this contruction for completeness (in the complex held). 
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Theorem 5.12 (HOSVD, De Lathauwer, De Moor, Vandewalle, |3]) 

A tensor A G ® ... (8) can be multiplied in the i-th mode by uni¬ 

tary matrices Ui G U{ni 1) in such a way that the resulting tensor S has 
the following properties: 

1. (i) (all-orthogonality) For any i = l,...,d and a = 0,...,ni denote 

by the slice in 8 ... ... 8 obtained by fixing the 

i-index egual to a. Then for 0 < a < fi < Ui we have Sf-S'^p = 0, that is 
any two parallel slices are orthogonal according to Hermitian product. 

2. (a) (ordering) for the Hermitian norm, for all i = 1,... ,d 

||sjti>!|s:||>,,,>||s‘.|| 

11 S'* 11 are the i-mode singular values and the columns of Ui are the i-mode 
singular vectors. For d = 2, US'*!! do not depend on i and we get the classical 
SVD. This notion has an efficient algorithm computing it. We do not pursue 
it further because the link with the critical points of the distance is weak, 
although it can be employed by suitable iterating methods 
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